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(54) Method of providing background patterns for camera tracking 

(57) An improved method of tracking a camera (14) 
in dynamic chroma-keying of the type in which a fore- 
ground object is photographed against a multicolored 
chroma-key screen. The multicolor patterns are created 
by projection (32,30) on a monochrome screen (20), for 
example by projecting light through a mask on a screen 
colored with the chroma-key color. The projection of 
multiple patterns can emulate the use of a single pattern 
having multiple levels of detail, and can enable lateral 
tracking of the camera. Sequentially projecting known 
transformations (for example, rotations) of a pattern 
adds an extra degree of freedom which further 
enhances the accuracy of the method. 
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Description 

FIELD AND BACKGROUNH OF THE INVENTION 

The present invention relates to video compositing s 
and. more particularly, to a method for providing a back- 
ground pattem so that the position and orientation of a 
moving camera can be determine in real lime for the 
realistic conrpositing of images of a foreground object 
with a virtual background. 

The technique of "chroma-key" compositing has 
long been used in video production to combine images 
of a foreground object with a virtual background stored 
in a digital data base. In this technique, the foreground 
object is photographed against a background of a is 
"chroma-key" color, typically a particular shade of blue. 
In the digital images thus captured, all the pixels colored 
with the chroma-key color are replaced with pixels 
derived from the virtual background. In order for the 
background pixels to be rendered in a way that provides 20 
a realistic composite image, the position and orientation 
of the camera nnust be known. 

Several method are known in the art for determin- 
ing the position and orientation of the camera. These 
methods can be classified in two categories. In methods 25 
of the first category, the position and orientation of the 
camera are tracked explicitly. In these methods, the 
camera must be providai with special tracking devices 
that are rigidly attached thereto. In some of the methods 
of the first category, the tracking devices are encoders. 30 
and lateral movement is tracked by encoders that are in 
physical contact with the floor of the studio. Another 
example of the methods of the first category a magnetic 
tracker system, is taught by Loftus et al., in PCT appli- 
cation no. US96/04846. Loftus et al.'s special tracking 35 
device is a magrietic tracker receiver rigidly attached to 
the camera. In the methods of the other category, the 
position and orientation of the camera are inferred from 
the captured images. For example. Graham, in US Pat- 
ent No. 5.502,482, which is incorporated by reference 40 
for all purposes as if fully set forth herein, teaches the 
use of a chroma-key background of two or more shades 
of blue in a predetermined pattern, for example a check- 
erboard pattern. The position and orientation of the 
camera are inferred from the locations of features of the 45 
pattern on the captured images. 

Methods such as Graham's have the advantage of 
allowing the use of conventional video cameras, without 
special tracking equipment. This is particularly conven- 
ient for video compositing with hand-held cameras, to so 
avoid the necessity of contacting lateral motion encod- 
ers with the floor of the studio, or to avoid the extra bulk 
and weight of trading equipment such as that of Loftus 
et al. Nevertheless, these methods have limitations of 
their own. One limitation is that the pattern is fixed, in ss 
both level of detail and location. If the camera zooms in 
too closely on the foreground object, there may be too 
few features in the portion of the image occupying the 


camera's field of view for the location and orientation of 
the camera to be determined accurately Conversely, if 
the camera is too far from the backgrourKl pattern, the 
pattern features may be sufficiently crowded to make it 
difficult to distinguish between a panning motion of the 
camera and a lateral translation. In principle, the pattern 
can be provided with multiple levels of detail, to pre- 
serve tracking resolution at all required distances; but 
this adds to the complexity of the pattern. Similarly if the 
foreground object is a live actor who moves laterally with 
respect to the pattern, and the camera follows the actor, 
the actor may move so far laterally that not enough of 
the pattern is left in the camera's field of view to allow 
the camera's location and orientation to be determined 
accuracy. A second limitation is that the multiplication of 
chroma-key colors has been found to degrade the real- 
ism of the composite image. The nan^ower the spectral 
band of the pixels that are replaced with virtual back- 
ground pixels, the more realistic the resulting composite 
image. The loss of realism associated with the use of a 
multi-shade chroma-key background persists even 
while the camera is in a fixed position and only one 
chroma-key color would suffice. A third limitation is that 
the pattem is fixed in place in the studio, typically being 
painted on a wall or a fixed partition. This limits the flex- 
ibility of the method and precludes its use, for example, 
in remote locations. 

There is thus a widely recognized need for. and it 
would be highly advantageous to have, a method for 
providing an image for dynamic chroma-key conposit- 
ing that is free of the limitations of the methods known in 
the art. 

SUMMARY OF THE INVENTION 

According to the present invention there is provided 
an improved video production method of the type in 
which a plurality of images of an object in a foreground 
volume are captured successively by a camera and 
superposed on a virtual background, the position and 
orientation of the camera t>eing determined using a first 
pattern featuring a first scale length and located at least 
partially beyond the object with respect to the camera, 
the improvement including the steps of: (a) positioning a 
screen at least partially beyond the object with respect 
to the camera; and (b) projecting the first pattern onto 
the screen. 

Simply stated, according to the present invention, 
the pattern is projected on a screen behind the fore- 
ground object. By "screen" is meant herein any suitable 
flat or curved surface on which the pattern may be pro- 
jected, although this surface preferably is flat. As is 
noted below, the screen may be either opaque or trans- 
parent The screen and the projector may be easily con- 
figured to be portable, enabling the method to be used 
in remote locations. In one prefered embodiment, the 
two colors of the pattern are two shades of a base color, 
with the screen being colored with the t>ase color and 
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the pattern being provided by projecting white light on 
the screen through a masK the shadow of the mask 
thus creating an area on the screen characterized by a 
darker shade of the base color than the part of the 
screen that is illuminated by the white light. If the cam- 
era is stationary for a prolonged period of time, or if a 
scene is shot in which a hand-held camera is not 
neoled, so that a camera equipped with one of the spe- 
cial explicit tracking devices described above may be 
used, or if a scene is shot in which the camera need not 
move at all. the projector is sinrtply turned off. The color 
of the screen then is used as a single chroma-key color, 
preserving the enhanced realism of the composite 
image that is provided by the use of only one chroma- 
key color. 

Multiple projectors are used to project multiple pat- 
terns on the screen. In this way, a pattern having multi- 
ple levels of detail is emulated. For example, if the 
camera moves so close to the screen that the level of 
detail of the first pattern is lost, a second pattern having 
a higher level of detail is superposed on the first pattern. 
Similarly, if the foreground object moves laterally, a sec- 
ond pattern is projected to the side of the first pattern, to 
provide continuous lateral coverage. 

BRIEF DESCRIPTION OF THE DRAWINGS 

The invention is herein described, by way of exam- 
ple only, with reference to the acconnpanying drawings, 
wherein: 

FIG. 1 is a schematic partial depiction of a video 
studio configured according to the present inven- 
tion; 

FIG. 2 illustrates the use of multiple projected pat- 
terns to emulate a single pattern having multiple 
levels of detail; 

FIG. 3 illustrates the use of multiple projected pat- 
terns to track a camera as the camera follows a lat- 
erally translating object. 

DESCRIPTION OF THE PREFERRED EMBODI- 
MENTS 

The present invention is of a method of providing 
background patterns which can be used to determine 
the position and orientation of a mobile camera during 
chroma-key compositing. 

The principles and operation of camera tracking 
according to the present invention may be better under- 
stood with reference to the drawings and the accompa- 
nying description. 

Referring now to the drawings. Figure 1 is a sche- 
matic partial illustration of a video studio configured 
according to the present invention. A camera 14 photo- 
graphs an object 1 0 against tiie background of a screen 
20. Object 1 0 is free to move within a foreground volume 
symbolized by dashed rectangle 12. Two projectors 30 


4 

and 32 are shown, each configured to project a pattern 
onto screen 20. Either but, not both projectors 30 and 
32 are used, each with a particular type of screen 20. If 
projector 30 is used, to project a pattern onto front sur- 

5 face 22 of screen 20 and obliquely relative to foreground 
volume 12, then screen 20 Is opaque. If projector 32 is 
used, to project a pattern onto rear surface 24 of screen 
20. then screen 20 is transparent, so that the pattern 
projected onto rear surface 24 is v^ible on the other 

10 side of screen 20. Note that light 31 tiiat is projected on 
to front surface 22 must be projected obliquely relative 
^ to foreground volume 12 to avoid projecting light onto 
object 10. Thus, if screen 20 is vertical, as shown In Rg- 
ure 1 , back-projecting from projector 32 has the advan- 

75 tage tiiat it is easier to obtain uniform illumination on 
screen 20 using light 33 that is projected substantially 
perpendicular to screen 20 than it is to obtain uniform 
illumination using obliquely projected light 31 . 

The pattern projected onto screen 20 is a geometric 

20 pattern chosen to facilitate the tracking of camera 14, 
/.e., the real time determination of the position and ori- 
entation of camera 14, from the images of the pattern 
behind object 10 as captured by camera 14. For exam- 
ple, the pattern may be a checkerboard, as descrbed in 

25 US Patent No. 5.502,482 cited above. More generally, 
the pattern is any pattern that enables the position and 
orientation of the camera to be inferred from the cap- 
tured images of the pattern. For example, the pattern 
may be a non-uniform checkertjoard. i.e., a grid of two 

30 mutually perpendicular sets of parallel lines in which the 
lines of each set are spaced non-uniformly. The compu- 
ter (not shown) that does the actual compositing is pro- 
vided with software to enable it to infer the position and 
orientation of camera 14, so that the background pixels 

35 in the images captured by camera 10 can be replaced 
realistically by virtual background pixels. For increased 
accuracy, at the start of a video session, the process is 
calibrated by providing the computer with initial values 
of the positional and orientationa! coordinates of cam- 

40 era 14. The first image captured by the camera then 
serves as a reference image, as described in the above- 
referenced US Patent No. 5,502,482. 

In general, each such pattern is characterized by a 
scale length. For example. In a non-uniform checker- 

45 board, the scale length is the average separation of two 
adjacent parallel lines. As noted above, the scale length 
of a pattern determines the range of distances between 
camera 14 and screen 20 over which that pattern can 
be used to track camera 14 accurately Figure 2 is a 

so frontal schematic illustration showing how multiple pro- 
jected patterns, characterized by two different scale 
lengths, are used to compensate for excursions of cam- 
era 14 outside this range of distances. In Rgure 2, the 
patterns are represented by circles 40 and 42. This rep- 

55 resentation is symtx>lic: a real pattern would have a suit- 
ably complicated geometry, such as the checkerboard 
descrbed above, within the area of the circle. Initially, 
pattern 40 is projected onto screen 20 to enable tiie 
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tracking of camera 14. tf camera 14 is about to move so 
dose to object 10 tliat pattern 40 has an insufficient 
level of detail to enable accurate tracking of camera 14, 
then a second pattern 42. having a scale length shorter 
than the scale length of pattern 40, Is projected onto 
screen 20. Typically, the scale length of pattern 42 is 
half the scale length of pattern 40. allowing camera 14 
to approach within half the distance to screen 20 that 
would t>e allowed if only pattern 40 were used. Prefera- 
bly, patterns 40 and 42 are projected simultaneously 
onto screen 22 during the capture of at least one image 
by camera 14. In this way, the inferred position of cam- 
era 14, based on pattern 40 at the closest allowed dis- 
tance of approach of camera 14 to screen 20 based on 
pattern 40, is used to calibrate the first image including 
pattern 42 as a new reference image. 

Conversely, if camera 14 is about to withdraw so far 
from screen 20 that the scale length of pattern 40 is too 
short to allow accurate tracking of camera 14, then a 
pattern with a longer scale length, typically twice the 
scale length of pattern 40. is projected onto screen 20 to 
allow camera 14 to withdraw at least twice as far from 
screen 20 as would have been allowed using only pat- 
tern 40. This set of pattems. including patterns 40 and 
42 and other patterns whose scale lengths are the scale 
length of pattern 40 multiplied or divided by powers of 
two, thus serves as a nested set of self-similar patterns, 
enabling camera 14 to be positioned over a much wider 
range of distances from screen 20 than is possible 
under the prior art methods. 

Figure 3 is a frontal schematic Illustration showing 
how multiple projected patterns are used to enable 
camera 14 to follow object 10 as object 10 moves later- 
ally with respect to screen 20. As in Figure 2, the pat- 
terns of Figure 3 are represented by circles 40 and 44. 
Initially, object 10 is in front of pattern 40. as seen from 
camera 14. If object 10 moves to the left, to the position 
of object 1 0', then not enough of pattem 40 is in the field 
of view of camera 14 to enable camera 14 to be tracked 
accurately Before object 10 reaches the position of 
object 10*. a second pattern 44 Is projected onto screen 
20. Pattern 44 is displaced far enough laterally from pat- 
tern 40 to allow the accurate tracking of camera 14 
when camera 14 is pointed at object 10'. As before, both 
patterns 40 and 44 are projected simultaneously onto 
screen 22 during the capture of at least one image by 
camera 14, to allow the first image of pattern 44 to be 
used as a reference image. 

As noted above, if screen 22 is opaque, then prefer- 
ably two different shades of the same base color are 
used as chroma-key colors, and front surface 22 is 
colored with the base color. The pattern (for example, 
pattern 40) is created on surface 22 by projecting the 
image of a mask onto front surface 22: the shadowed 
areas then are the portion of the pattern that is colored 
with a darker shade of the base color, and the rest of the 
pattern retains the lighter shade of the base color. The 
light used to project the image of the mask may be 


white, or may be the color of front surface 22. 

If the projected pattern is altered dynamically under 
computer control, then another degree of freedom is 
provided to enhance the accuracy of the tracking of 

5 camera 1 4. This can be done, for example, using one of 
the projectors manufactured by BARCO Projection Sys- 
tems of Kuurne, Belgium. Specifically, the patterns pro- 
jected onto screen 20 subsequent to the first pattern are 
transformed replicas of the first pattern. For example, a 

10 subsequent pattern may be a replica of the first pattern 
translated by a known amount, a replica of the first pat- 
tern dilated or contracted by a known amount, a replica 
of the first pattern subjected to a known affine transfor- 
mation, or a replica of the first pattern rotated by a 

15 known amount. Preferably, the patterns are projected 
sequentially in coordination with image capture by cam- 
era 14, with each transformed replica projected onto 
screen 20 during the entire capture by camera 14 of one 
or more images of object 10 against the background of 

20 the pattem. so that the pattern is stable on the caiptured 
image. The extra degree of freedom provided by this 
dynamic alteration of the pattern conpensates to a cer- 
tain extent for loss of resolution due to camera 14 being 
too close to screen 20 or too far from screen 20 to be 

25 tracked accurately relative to a static pattern of a given 
level of detail. For example, suppose the pattern rotates 
at a known uniform angular velocity. The changes in the 
positions of pattern features from one captured image to 
the next are functions, not only of the positional and ori- 

30 entational coordinates of camera 14. but also of the 
radial distances of those features from the center of 
rotation on screen 20. This provides information equiva- 
lent to the provision of a second pattem having a 
shorter scale length than the first pattern, so that a 

35 rotating pattern can be used to track camera 14 at 
closer distances from screen 20 than a static pattern. 

If the transformation applied to the pattern is suffi- 
ciently simple, for example a rotation or a periodically 
oscillating translation, then the transformation may be 

40 effected by mechanical means, for example, by physi- 
cally rotating a "Gobo" mask inside a "Moving Light" 
projector. 

While the invention has been described with 
respect to a limited number of emtxxiiments, it will be 

45 appreciated that many variations, modifications and 
other applications of the invention may be made. 

\A/here technical features mentioned in any daim 
are followed by reference signs, those reference signs 
have been included for the sole purpose of increasing 

so the intelligibility of the daims and accordingly, such ref- 
erence signs do not have any limiting effect on the 
scope of each element identified by way of example by 
such reference signs. 

55 Claims 

1 . An improved video production method of the type in 
which a plurality of images of an object in a fore- 
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ground volume are captured successively by a 
camera and superposed on a virtual background, 
the position and orientation of the camera being 
determined uang a first pattern featuring a first 
scale length and located at least partially beyond 5 
the object with respect to the camera, the improve- 
ment comprising the steps of: 

(a) positioning a saeen at least partially 
beyond the object with respect to the camera; 10 
and 

(b) projecting the first pattern onto said screen. 

2. The method of claim 1 , wherein said projection is 
effected obliquely with respect to said foreground is 
volume. 


tern is translated relative to said first pattern. 

13. The method of claim 10. wherein said second pat- 
tern is dilated relative to said first pattern. 

14. The method of claim 10, wherein said second pat- 
tern Is contracted relative to said first pattern. 

15. The method of claim 10. wherein said second pat- 
tern is an affine transformation of said first pattern. 

n 

- 16. The method of claim 10, wherein laid second pat- 
tern is rotated relative to said first pattern. 


3. The method of claim 1, wherein said screen is 
transparent, and wherein said projection is effected 
from behind said screen w/Hh respect to said object. 20 

4. The method of claim 1 , wherein said screen has a 
surface featuring a chroma-key color, and wherein 
said projection is effected by directing light through 

a mask. 2$ 

5. The method of claim 4, wherein said light is sub- 
stantially white. 

6. The method of claim 1, further comprising the step so 
of: 


(c) projecting a second pattern onto the screen. 

7. The method of daim 6, wherein said second pat- 35 
tern features a second scale length different from 
the first scale length. 

8. The method of daim 6, wherein said second pat- 
tern is displaced laterally with respect to the first 40 
pattern. 

9. The method of claim 6, wherein the first pattern and 
said second pattern are projected on the screen 
together at least for the duration of the capture of 45 
one of the images of the object. 


10. The method of daim 6, wherein said second pat- 
tern is a transformed replica of the first pattern. 

so 

1 1 . The method of claim 10, wherein the first pattern is 
projected during the capture of a first image of the 
object, and wherein said second pattern is pro- 
jected during the capture of a second image of the 
object immediately subsequent to the capture of the ss 
first image of the object. 

12. The method of claim 10. wherein said second pat- 
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(54) Method of providing background patterns for camera traddng 


(57) An improved method of tracking a camera (14) 
in dynamic chroma-keying of the type in which a fore- 
ground object is photographed against a multicolored 
chroma-key screen. The multicolor patterns are created 
by projection (32.30) on a monochrome screen (20), for 
example by projecting light through a mask on a screen 
colored with the chroma-key color The projection of 


multiple patterns can emulate the use of a single pattern 
having multiple levels of detail, and can enable lateral 
tracking of the camera. Sequentially projecting known 
transformations (for example, rotations) of a pattern 
adds an extra degree of freedom which further 
enhances the accuracy of the method. 
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